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1.0  INTRODUCTION  AND  SUMMARY 

This  document  summarizes  work  performed  by  Delta  Information  Systems,  Inc. 

(DIS)  for  the  National  Communications  System  (NCS),  Office  of  Technology  and 
Standards.  This  office  is  responsible  for  the  management  of  the  Federal 
Telecommunications  Standards  Program,  which  develops  telecommunications  standards, 
whose  use  is  mandatory  for  all  Federal  departments  and  agencies.  This  study  was 
performed  under  task  order  number  88-009  of  contract  number  DCA100-87-C-0078. 

This  report  covers  investigations  and  measurements  of  tempot  jI  frequency 
response  as  a  means  to  numerically  analyze  the  motion  performance  capability  of  a  digital 
video  teleconferencing  codec.  This  performance  parameter  was  previously  introduced 
under  task  order  number  87-010  of  the  same  contract.  The  effort  described  in  this  report 
contains  a  more  detailed  analysis,  refinement  of  measuring  techniques  and  a  large 
selection  of  measured  values  of  temporal  frequency  response  and  other  related 
performance  parameters  on  three  different  codec  models.  The  techniques  described  herein 
provide  a  complete  objective  analysis  of  moving  pictures  which  is  an  important  factor  in 
the  performance  evaluation  of  any  digital  video  teleconferencing  system. 

Section  2.0  gives  the  background  of  previous  NCS  programs  which  are  directly 
related  to  this  effort.  Section  3.0  reviews  the  parameters  which  can  be  used  to  describe 
the  motion  performance  of  a  digital  video  codec,  and  the  related  artifacts  which  can  be 
observed  in  the  picture,  it  also  establishes  the  requirements  for  a  practical  method  to 
evaluate  such  artifacts.  Section  4.0  covers  the  development  of  the  test  procedure  in 
accordance  with  these  requirements.  It  reviews  the  test  signals  and  describes  the 
methodology  of  their  application,  giving  details  of  the  test  setup  and  the  standard  test 
equipment  used  in  the  process.  The  test  results  are  summarized  in  Section  5.0,  covering 
temporal  frequency  response  and  related  parameters.  The  results  are  illustrated  by  typical 
examples  of  each  covered  parameter.  Section  6.0  contains  a  discussion  of  the  test  results 
and  shows  them  to  be  fully  logical  and  in  accordance  with  expectations  and  limited 
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observations.  Section  7.0  gives  a  brief  review  of  the  program  and  contains 
recommendations  for  further  efforts  in  the  objective  performance  eva'uation  of  digital 
video  codecs. 

2.0  BACKGROUND 

2.1  Previous  NCS  Programs 

Two  closely  related  previous  tasks  of  the  same  overall  NCS  program  form  a  direct 
basis  for  the  ongoing  investigation  of  the  applications  of  temporal  frequency  response 
measurements.  They  are  Task  No.  87-011,  Development  of  a  Video  Tape  to  Correlate 
Subjective  and  Objective  Testing  of  Teleconference  System  (Final  Report  dated  May  18, 
1990),  and  Task  No.  87-010,  Standardization  of  End-to-End  Performance  for  Full  Motion 
Video  Teleconferencing  (Final  Report  dated  May  21,  1990).  Both  these  tasks  in  turn  use 
background  material  prepared  in  previous  years  as  part  of  other  NCS  programs. 

Task  No.  87-01 1  covers  the  steps  taken  in  the  development  of  a  tape  for  objective 
testing  of  motion  performance.  The  Final  Report  gives  the  pertinent  data  of  the  various 
test  patterns  and  the  background  for  their  application  to  the  measurement  of  temporal 
frequency  response.  Task  No.  87-010  includes  details  of  the  design  and  development  of 
the  test  patterns.  It  shows  preliminary  measurement  results  and  sample  graphs  of 
temporal  frequency  responses.  The  results  are  generally  as  expected  and  logical  but 
indicate  that  the  test  methodology  needs  refinement  to  achieve  more  consistency.  Such 
refinements  are  addressed  in  the  task  on  hand. 

2.2  T 101. 5  Committee  Contributions 

The  Subworking  Group  on  Video  Teleconferencing A/'ideo  Telephony  of  the 
American  National  Standards  Institute  (ANSI)  Committee  T1Q1.5  is  developing  Analog 
Interface  Performance  Specifications  for  Digital  Video  TeleconferencingA/ideo  Telephony 
Service.  During  this  process,  numerous  contributions  were  submitted  by  experts  from 
various  organizations  covering  motion  performance  degradations  and  possible  methods  of 
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their  measurement.  These  contributions  furnish  a  thorough  overview  of  observed  motion 
artifacts  and  provide  material  for  the  analysis  of  their  relation  to  temporal  frequency 
response.  This  makes  it  possible  to  judge  the  utility  of  temporal  frequency  response 
measurements  for  the  overall  evaluation  of  the  ability  of  a  teleconferencing  system  to 
reproduce  motion. 

3.0  MOTION  PERFORMANCE  PARAMETERS 
3.1  Spatial  and  Temporal  Performance 

A  digital  video  teleconferencing  system  requires  establishment  of  more  performance 
parameters  than  an  analog  system.  Conventional  parameters  used  for  instance  in  EIA  RS- 
250B  or  ANSI  T1.502  are  spatial  which  means  that  they  can  be  evaluated  with  stationary 
signals.  Most  additional  parameters  are  temporal  and  concerned  with  motion  rendition. 

New  definitions  and  motion  test  signals  are  being  developed  to  evaluate  temporal 
performance. 

A  very  simplified  block  diagram  of  a  digital  video  system  is  shown  on  Figure  3.1 .  It 
contains  the  main  system  elements  that  determine  transmission  quality  and  indicates  the 
sections  which  influence  spatial  and  temporal  performance. 

The  NTSC  decoder  and  encoder  determine  essentially  all  spatial  performance 
parameiers.  The  design  of  these  circuits  is  well  developed  and  normally  produces  flawless 
performance.  Linearity  is  generally  no  problem  since  most  elements  producing  non- 
linearities  in  analog  systems  are  not  present.  Spatial  performance  of  codecs  is  largely 
independent  of  encoding  algorithms  and  bit  rate  and  in  most  cases  meets  or  at  least 
approaches  high  quality  video  specifications.  This  has  been  proven  by  conventional  tests 
on  several  different  codecs  over  the  full  range  of  bit  rates.  The  only  important  exception 
to  this  statement  is  spatial  frequency  response  which  is  controlled  by  the  bandwidth  of  the 
filters  on  the  transmit  and  receive  side  which  must  be  matched  to  the  sampling  rate  of  the 
digital  .'-'^coder  and  thus  is  indirectly  dependent  on  the  transmission  bit  rate. 


TYPICAL  DIGITAL  VIDEO  SYSTEM 


Temporal  performance  parameters  are  determined  by  the  digital  system  elements 
namely  encoder,  decoder  and  buffers  which  are  typical  for  all  interframe  coding 
techniques.  Numerous  artifacts  are  produced  by  limitations  in  the  encoding  algorithm  and 
buffer  size.  A  typical  motion  artifact  catalogue  is  contained  in  Document  T1Q1. 5/90-106. 
Many  test  methods  have  been  proposed  which  generally  measure  one  specific  artifact. 
Such  tests  are  to  yield  numerical  results  which  can  be  correlated  with  subjective 
evaluation. 

In  spite  of  the  considerable  past  effort  to  define  and  measure  motion  artifacts,  there 
has  been  no  consensus  between  the  many  involved  parties.  The  main  problem  is  that 
different  encoding  algorithms  often  produce  artifacts  which  are  only  slightly  different  and 
thus  difficult  to  distinguish.  The  exact  wording  of  artifact  definitions  also  causes  many 
differences  of  opinion.  The  following  artifact  list  is  a  composite  of  several  contributions 
edited  to  fit  into  typical  distinct  categories. 

A.  Resolution  Degradation  •  The  deterioration  of  motion  video  such  that  the 
received  video  imagery  has  suffered  a  loss  of  temporal  resolution  and 
response.  It  is  largely  caused  by  Temporal  Slope  Overload  which  appears 
mainly  in  predictive  coding  schemes.  It  is  generated  when  the  prediction 
error  signal  increases  and  exceeds  the  greatest  representative  value  o^  the 
quantizing  characteristics. 

Blurring/Smearing  -  The  deterioration  of  a  motion  video  picture  in 
which  the  received  image  has  lost  edges  and  detail  of  the  original 
imagery.  Fast  moving  objects  may  appear  completely  wiped  out. 

Edge  Busyness  -  The  deterioration  of  motion  video  such  that  the 
outlines  of  moving  objects  are  displayed  with  randomly  varying 
activity. 

Loss  of  Contrast  •  The  received  video  imagery  is  unable  to  reproduce 
the  contrast  range  of  the  original  because  of  temporal  slope 
limitations.  The  picture  has  a  "washed  out"  appearance. 

B.  Spurious  Signals  -  The  received  video  imagery  contains  elements  not  present 
in  the  original. 

Blocking  -  The  received  video  imagery  possesses  rectangular  or 
checkerboard  patterns  not  present  in  the  original. 
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Mosquito  Noise  -  The  quantizing  noise  generated  by  the  block 
processing  of  moving  objects  that  gives  the  appearance  of  false  small 
moving  objects  (e.g.  a  mosquito  flying  around  a  person's  head  and 
shoulders). 

Dirty  window  -  Din/  window  is  a  kind  of  granular  noise  which 
remains  after  all  objects  have  gone  away.  It  occurs  when  conditional 
picture  element  replenishment  is  used  in  coding  based  on  interframe 
correlations.  When  threshold  values  are  made  too  high  to  control  the 
effective  number  of  picture  elements  across  the  input  video,  part  of 
the  noise  in  the  picture  remains,  so  it  seems  as  though  one  is  looking 
into  a  room  through  a  dirty  window. 

C.  Jerkiness  *  The  continuous  motion  in  the  original  picture  is  disturbed.  The 
received  picture  may  show  any  degree  of  motion  deterioration  from  a  slight 
loss  of  smoothness  to  a  "series  of  distinct  snapshots". 

D.  Image  Persistence  -  The  appearance  of  parts  of  earlier  video  frames  of  a 
moving  and/or  changing  object  within  the  current  video  frame. 

Erasure  *  An  object  that  was  erased  continues  to  appear  in  the 
received  video  imagery. 

Retention  •  After  a  frame  cut,  distinct  portions  of  the  previous  image 
remain  visible  for  several  frames.  It  takes  several  frames  to  build  up 
the  new  image  to  full  amplitude. 

3.2  Artifact  Evaiuation 

In  spite  of  frequent  similarities  of  the  various  artifacts  listed  above,  each  of  them 
has  some  distinct  features  which  can  be  extracted  and  measured.  This,  however,  is  much 
more  easily  said  than  done.  A  good  overview  of  potentially  applicable  techniques  is  given 
in  ANSI  Committee  T1  Contribution,  Document  Number  T1Q1. 5/90-108,  entitled  "Feature 
Extraction  for  Automated  Quality  Assessment  of  Digitally  Transmitted  Video"  dated  Jan. 
30,  1 990.  Many  proposed  techniques  depend  on  the  simultaneous  availability  of  the 
original  (undistorted)  picture  and  the  digitally  processed  (distorted)  codec  output  picture. 
Such  a  requirement  represents  a  serious  and  often  unsurmountable  constraint  since  in 
practice  most  measurements  must  be  made  on  an  installed  system  and  not  on  equipment 
connected  back-to-back  in  a  laboratory.  Implementation  of  the  proposed  feature  extraction 
and  artifact  measurement  techniques  can  be  fairly  complex  and  requires  sophisticated  test 
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equipment. 

The  user  of  a  video  teleconferencing  system  is  concerned  with  the  overall 

appearance  and  acceptability  of  the  received  picture  and  not  with  the  exact  value  of  a 

specific  artifact.  That  is  particularly  true  since  many  artifacts  are  so  similar  that  only  a 

digital  video  expert  is  able  to  distinguish  them.  Therefore,  it  would  be  desirable  for  a 

system  to  measure  the  ability  of  a  teleconferencing  system  to  reproduce  motion  to  have 

the  following  properties. 

Firmly  defined  and  easily  applied  test  signals 

Numerical  results  describing  system  performance 

Ability  to  perform  measurements  when  only  the  processed  picture  is 
available 

Stability  and  repeatability  of  test  results 

Ability  to  measure  multiple  performance  parameters  and  artifacts  with  the 
same  test  signal 

Ability  to  determine  numerical  limits  of  acceptability  for  various  system 
applications 

Ease  of  numerical  evaluation 

Though  almost  any  picture  can  be  used  to  test  digital  video  performance 
subjectively,  the  need  for  consistent  repeatability  and  numerical  evaluation  calls  for  an 
artificially  generated  test  pattern  for  objective  tests.  Such  a  pattern  can  be  produced  by  a 
number  of  available  computer  video  generation  equipments.  The  ultimate  solution  will  be  a 
specially  designed  test  pattern  generator  but  more  research  is  needed  to  make  the  design 
of  such  a  unit  practical. 

4.0  TEST  PROCEDURE 
4.1  Concept 

Modern  technology  has  eliminated  motion  as  a  significant  fc.ctor  in  standard 
broadcast  TV.  Therefore,  all  conventional  test  signals  for  evaluation  of  broadcast  TV 
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performance  are  static.  However,  the  large  amount  of  bit  rate  reduction  necessary  in 
digital  video  teleconferencing/video  telephony  service  makes  use  of  the  frame-to-frame 
redundancy  of  picture  content  which  in  turn  affects  the  fidelity  of  motion  rendition.  This 
calls  for  a  test  technology  utilizing  time  variant  test  signals. 

Motion  measurements  should  follow  the  well  established  techniques  of  static  video 
measurements  as  much  as  feasible.  A  prime  static  performance  parameter  is  spatial 
frequency  response  which  can  be  defined  as  the  dependence  of  video  amplitude  on 
different  spacings  of  black  and  white  picture  elements  as  produced  by  the  conventional 
resolution  test  chart  or  a  video  sweep.  Its  temporal  equivalent  is  temporal  response  which 
can  be  defined  as  the  ability  of  a  television  system  to  respond  to  periodic  changes  (varying 
periodicity)  from  frame-to-frame  (the  time  domain). 

However,  while  the  measurement  of  spatial  frequency  response  is  generally 
straightforward  and  independent  of  the  test  signal,  the  techniques  of  digital  video  make 
the  measurement  of  temporal  frequency  response  more  complex.  The  stress  on  the  codec 
algorithm  depends  on  the  amount  of  detail  in  the  whole  picture,  and  the  temporal  response 
will  change  accordingly.  Thus,  the  same  temporal  frequency  may  produce  different 
response  values  with  test  patterns  containing  different  amounts  of  detail. 

Most  motion  artifacts,  as  listed  in  para.  3.1,  result  in  distortions  of  the  produced 
video  amplitude.  A  subjective  evaluator  is  not  inclined  to  separate  many  of  these  often 
very  similar  artifacts  but  judges  the  picture  quality  based  on  the  integrated  total 
impression.  Since  temporal  response  measures  video  amplitude  distortion,  it  also  gives  a 
composite  of  the  effects  of  most  of  these  artifacts.  It  does  not  require  complex  extraction 
of  individual  features  followed  by  a  sophisticated  mathematical  analysis  but  provides  a 
practical  objective  measure  of  the  integrated  offect  of  a  wide  range  of  motion 
deteriorations,  thus  simulating  what  the  observer's  eye  sees  during  subjective  tests. 

One  temporal  cycle  is  produced  by  one  black-white-black  transition  at  one  picture 
element  in  the  moving  test  pattern,  either  by  motion  or  by  switching.  This  transition 


ideally  generates  a  square  wave  which  is  degraded  after  passing  through  a 
teleconferencing  video  codec.  This  degradation  manifests  itself  as  a  reduced  or  otherwise 
distorted  slope  and/or  loss  of  peak-to-peak  amplitude  of  the  transition.  Both  of  these 
factors  can  be  readily  measured  by  the  RMS  value  of  the  resulting  waveform  amplitude. 

4.2  Implementation 

Any  motion  in  a  picture  results  in  a  change  of  pixels  in  consecutive  frames.  Motion 
artifacts  occur  when  the  amount  of  change  is  such  that  the  codec  cannot  reproduce  it 
faithfully.  In  order  to  achieve  repeatable  numerical  results,  the  pixel  change  must  be 
implemented  with  firmly  defined  patterns,  it  can  be  produced  either  by  actual  motion  of 
the  same  pattern  or  by  switching  between  two  different  patterns  (scene  cut). 

Two  different  motion  test  patterns  were  generated  on  a  Cubicomp  computer 
controlled  special  effects  equipment  and  recorded  on  video  tape.  The  first  was  a  simple 
scene  cut  while  the  second  employed  a  rotating  wheel  pattern.  The  scene  cut  pattern  was 
produced  by  switching  between  white  circles  on  a  black  background  and  a  black  field. 

Size  and  spacing  of  the  circles  and  the  rate  of  switching  are  variable,  forming  32  different 
scene  cut  test  patterns.  They  result  in  pixel  changes  of  about  60%  and  30%  at  each 
switching  and  temporal  frequencies  between  0.125  and  7.5  cycles  per  second. 

The  rotating  wheel  pattern  has  two  parts,  namely  a  static  background  with  a  white 
center  section  and  segments  of  bar  chart  colors  around  the  perimeter,  and  a  rotating 
overlay  consisting  of  black  spokes.  These  spokes  have  3  different  width,  each  using 
between  6  and  9  rotation  speeds,  forming  23  different  rotating  wheel  test  patterns.  These 
patterns  produce  pixel  changes  of  between  2.2%  and  25%  per  frame  and  temporal 
frequencies  of  between  0.33  and  3.75  cycles  per  second.  One  of  the  patterns  is  shown 
on  Figure  4.1. 

Full  details  on  the  test  patterns  are  contained  in  the  previously  submitted  final 
reports  referenced  in  para.  2.1  of  this  report.  For  ease  of  reference,  the  basic  parameters 
of  all  implemented  patterns  are  listed  on  Tables  4-1  and  4-2.  The  fastest  switching  rate 


9 


ROTATING  OVERLAY 


FIGURE  4.1 
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SCENE  CUT  PATTERNS 


ROTATION  SPEED 


TEST 

PATTERN 

NO. 

SPOKE 

WIDTH 

(DEGREES) 

FRAMES/ 

REVOLUTION 

DEGREES/ 

SECOND 

TEMPORAL 

FREQUENCY 

(CPS) 

FRAMES/ 

SPOKE 

%  PIXEL 
CHANGE/ 
FRAME 

%  BLOCK 
CHANGE/ 
FRAME 

1 

30 

540 

20 

0.33 

45 

2.2 

18 

2 

30 

360 

30 

0.50 

30 

3.3 

3 

30 

240 

45 

0.75 

20 

5.0 

4 

30 

180 

60 

1.00 

15 

6.7 

22 

5 

30 

144 

75 

1.25 

12 

8.3 

6 

30 

120 

90 

1.50 

10 

10.0 

7 

90 

120 

2.00 

7.5 

13.3 

34 

8 

30 

72 

150 

2.50 

6 

16.7 

9 

30 

60 

180 

3.00 

5 

20.0 

43 

10 

18 

720 

15 

0.42 

36 

2.8 

31 

11 

18 

540 

20 

0.55 

27 

3.7 

12 

18 

360 

30 

0.85 

18 

5.6 

13 

18 

240 

45 

1.25 

12 

8.3 

14 

18 

180 

60 

1.67 

9 

11.1 

42 

15 

18 

144 

75 

2.10 

7.2 

13.9 

16 

18 

120 

90 

2.50 

6 

16.7 

17 

18 

90 

120 

3.33 

4.5 

22.2 

57 

18 

10 

720 

15 

0.75 

20 

5.0 

50 

19 

10 

540 

20 

1.00 

15 

6.7 

54 

20 

in 

360 

30 

1.50 

10 

10 

21 

10 

240 

45 

2.25 

6.7 

15 

22 

10 

180 

60 

3.00 

5 

20 

70 

23 

10 

144 

75 

3.75 

4 

25 

75 

TABLE  4-2 

ROTATING  WHEEL  PATTERNS 
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(highest  temporal  frequency)  on  Table  4*1  is  not  usable  in  practice  because  all  codecs  use 
frame  repetition  rates  of  2  or  higher  which  wipes  out  the  fastest  switching. 

The  test  patterns  are  recorded  on  3/4"  tape  and  processed  through  codecs  of 
different  manufacture  operating  at  various  bit  rates  covering  the  full  range  of  64  to  1 544 
Kbps.  The  outputs  are  again  recorded  on  3/4"  tape  which  is  analyzed  in  the  test  setup 
shown  on  Figure  4.2.  The  VO-9850  Tape  Recorder  allows  manual  tape  advance  and 
identification  of  each  frame  by  means  of  a  frame  counter.  The  KM-F250  Frame 
Synchronizer  re-generates  the  degraded  synch  waveform  of  the  still  picture  from  the  tape 
recorder  and  makes  it  compatible  with  the  \/M-700  Video  Measurement  Set.  This 
equipment  allows  amplitude  measurement  at  any  exactly  defined  point  in  the  picture  with 
an  accuracy  of  better  than  one  IRE  unit.  It  also  makes  it  possible  to  recognize  all  temporal 
advances  of  the  test  pattern.  An  amplitude  measurement  is  made  after  every  advance 
even  if  there  has  been  no  change  in  amplitude.  Measurements  are  taken  over  about  4  to 
10  temporal  cycles,  depending  on  pattern  detail,  rotation  or  switching  speed  and  the 
amount  of  recognized  amplitude  variations. 

Several  series  of  tests  were  performed,  using  both  types  of  patterns  through 
various  codecs.  During  the  performance  of  these  tests,  it  became  apparent  that  the 
results  obtained  with  the  rotating  wheel  pattern  were  more  consistent  than  those  with  the 
scene  cut.  Furthermore,  the  rotating  wheel  patterns,  though  artificial,  more  closely 
simulate  actual  motion  which  makes  them  preferable.  In  addition,  the  motion  simulated  by 
a  scene  cut  is  unique  and  various  codec  algorithms  may  respond  to  it  quite  differently.. 

The  scene  cut  pattern  may  unduly  favor  one  codec  design  over  another.  It  therefore  was 
decided  to  use  only  the  rotating  wheel  pattern  for  further  temporal  frequency  response 
measurements,  though  the  scene  cut  pattern  turned  out  to  be  useful  for  other  purposes. 

The  rotating  wheel  patterns  as  originally  designed  aopeared  to  be  adequate  for  the 
range  of  codecs  and  bit  rates  to  be  tested  and  have  been  left  unchanged  for  further  tests. 
However,  the  test  instrumentation  was  modified  to  remove  possible  ambiguities  and 
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TEST  SETUP  -  TEMPORAL  FREQUENCY  RESPONSE 


improve  convenience  and  accuracy.  Originally,  the  measurement  was  made  on  the  line 
passing  through  the  center  of  the  picture  (approximately  line  1 40)  but  at  this  location  there 
often  is  a  difference  between  the  two  fields  of  one  frame  just  at 
the  critical  moment  of  the  white-black  transition  which  produces  an  ambiguity  in  signal 
level.  Making  the  measurement  near  the  center  (approximately  36  microseconds  after  the 
start  of  horizontal  blanking)  of  a  line  near  the  top  of  the  white  portion  (approximately  line 
70,  exact  location  not  critical)  as  shown  on  Figure  4.1,  eliminates  this  problem. 

All  codecs  use  frame  repetition  to  limit  the  total  amount  of  data  to  be  transmitted. 
This  reduces  the  frame  rate  to  1 5  or  below  but,  depending  on  type  and  use  of  the  picture, 
is  largely  not  seriously  objectionable.  Due  to  this  repetition,  it  is  not  necessary  to  measure 
the  signal  level  of  every  frame,  only  whenever  an  actual  frame  advance  occurs.  These 
points  can  easily  be  identified  using  the  cursor  on  the  \/M-700  display,  providing  enhanced 
convenience  and  accuracy.  Printouts  of  two  consecutive  typical  displays  are  shown  on 
Figures  4.3  and  4.4,  demonstrating  that  a  frame  advance  is  easily  recognizable. 

The  measured  levels  are  processed  in  a  computer,  resulting  in  a  printout  a  sample 
of  which  is  shown  on  Figure  4.5.  It  contains  the  graph  of  signal  level  vs.  frame  number, 
the  RMS  value  of  the  signal  levels  and  the  averrge  distance  between  frame  advances, 
giving  the  effective  frame  repetition  rate.  The  RMS  value  provides  one  point  on  a  temporal 
response  curve  derived  from  test  patterns  of  one  spoke  width  at  different  rotation  speeds. 
With  increasing  rotation  i^peed  and  temporal  frequency,  the  rrame  repetition  rate  increases 
provided  the  codec  algorithm  has  an  adaptive  repetition  rate  which  is  common  with  low  bit 
rate  codecs. 

In  cases  of  a  severe  challenge  imposed  by  the  test  pattern  on  the  codec  algorithm, 
buffer  overload  may  produce  additional  distortions  toward  the  bottom  of  the  picture.  This 
is  easily  recognized  and  measured  using  the  line  selection  feature  of  the  VM-700.  If 
desired,  a  pattern  analysis  on  about  line  210  gives  a  "picture  bottom"  e<^uivalent  to  the 
usual  measurement  on  line  70. 
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VM-700  DISPLAY  AND  AMPLITUDE  MEASUREMENT 


FIGURE  4.4 

VM-700  DISPLAY  AND  AMPLITUDE  MEASUREMENT 
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FIGl’^E  4.5 

PRINTOUT  OF  WAVEFOltM,  RMS  &  FRAME  REPETITION  RATE 


Numerical  measurement  results  and  discussions  are  contained  in  the  following 
sections. 

5.0  TEST  RESULTS 
5.1  Temporal  Response 

The  rotating  wheel  patterns  the  details  of  which  are  given  on  Table  4-2  were 
processed  through  3  different  codecs  and  recorded  on  3/4"  video  tape.  The  codecs  are 
designated  H,  L,  and  P.  Codec  H  covers  the  range  of  384  to  1 544  Kbps  while  codecs  L 
and  P  operate  over  the  range  of  64  to  384  Kbps.  The  operating  bit  rates  were  selected 
from  the  P  x  64  hierarchy  and  are  64,  1 28,  256,  and  384  Kbps  for  codecs  L  and  P,  and 
384,  768,  and  1536  Kbps  for  codec  H. 

Initial  visual  inspection  showed  that  due  to  the  wide  range  of  bit  rates  not  all  test 
patterns  produced  significant  and  usable  results.  Patterns  with  wide  spokes  and  low 
speeds  do  not  provide  sufficient  challenge  at  high  bit  rates  while  narrow  spokes  and  high 
speeds  generally  far  exceed  the  capability  of  low  bit  rate  codecs.  Therefore,  specific 
groups  of  patterns  were  selected  for  numerical  evaluation  of  each  bit  rate  and  codec.  A 
sample  of  such  a  selection  is  shown  on  Table  5-1  which  is  simply  a  marked-up  copy  of 
Table  4-2. 

Using  the  test  setup  of  Figure  4.1,  an  amplitude  measurement  is  made  following 
each  test  pattern  advance.  The  results  are  recorded  on  a  data  sheet,  a  filled-in  sample  of 
which  is  shown  on  Table  5-2.  The  frame  numbers  and  amplitudes  are  processed  in  a 
computer,  resulting  in  the  previously  mentioned  printout,  a  sample  of  which  is  shown  on 
Figure  4.5.  The  RMS  amplitude  number  on  this  figure  represents  one  value  of  temporal 
response.  Analyzing  various  rotation  speeds  of  the  same  test  pattern  under  identical  test 
conditions  results  in  a  series  of  points  on  a  temporal  response  curve.  As  indicated  on 
Table  5-1,  portions  of  two  different  spoke  width  patterns  were  analyzed  for  every  bit  rate 
on  every  codec.  Thus,  as  shown  on  Figure  5.1 ,  two  temporal  response  curves  can  be 


19 


ROTATION  SPEED 


TEST 

PATTERN 

NO. 

SPOKE 

WIDTH 

(DEGREES) 

FRAMES/ 

REVOLUTION 

DEGREES/ 

SECOND 

TEMPORAL 

FREQUENCY 

(CPS) 

FRAMES/ 

SPOKE 

%  PIXEL 
CHANGE/ 
FRAME 

%  BLOCK 
CHANGE/ 
FRAME 

1 

30 

540 

20 

0.33 

45 

2.2 

18 

2 

30 

360 

30 

0.50 

30 

3.3 

3 

30 

240 

45 

1 

1 

0.75 

20 

5.0 

j 

4 

30 

180 

60 

1.00 

15 

6.7 

22 

5 

30 

144 

75 

1.25 

12 

8.3 

6 

30 

120 

90 

1.50 

10 

10.0 

7 

30 

90 

120 

2.00 

7.5 

13.3 

34 

8 

30 

72 

150 

2.50 

6 

16.7 

9 

30 

60 

180 

3.00 

5 

20.0 

43 

10 

18 

720 

15 

0.42 

36 

2.8 

31 

ll" 

18 

540 

20 

0.55 

27 

3.7 

12 

18 

360 

30 

0.85 

18 

5.6 

13 

18 

240 

45 

1.25 

12 

8.3 

14  ! 

18 

180 

60 

1.67 

9 

11.1 

42 

i 

15  ! 

1 

18 

144 

75  i 

2.10 

7.2 

13.9 

1 

1 

16 

18 

120 

90 

2.50 

6 

16.7 

1 

1 

17 

18 

90 

120 

3.33 

4.5 

22.2 

57 

18 

10 

720 

15 

0.75 

20 

5.0 

50 

19 

10 

540 

20 

1.00 

15 

6.7 

54 

20 

10 

360 

30 

1.50 

10 

10 

21 

10 

240 

45 

2.25 

6.7 

15 

22 

10 

180 

60 

3.00 

5 

20 

70 

23 

10 

144 

75 

3.75 

4 

25 

75 

TABLE  5-1 

ROTATING  WHEEL  PATTERNS  SELECTED 
FOR  CODEC  L  AT  128  KBPS 
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CODEC:  L  RATE:  128  PATTERN:  3  LINE:  70  POS:  3 6 /MICROSEC 


TABLE  5-2 

AMPLITUDE  MEASUREMENT  DATA  SHEET 
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AMPL 


TEMPORAL  FREQUENCY 


drawn  for  every  codec/bit  rate  combination.  This  not  only  demonstrates  the  dependence 
of  the  results  on  the  test  pattern,  but  helps  in  verification  of  the  validity  of  the  test 
method.  It  does  not  seem  necessary  to  draw  all  such  possible  curves  but  the  range  of 
results  is  summarized  on  Tables  5-3,  5-4,  and  5-5. 

Codec  H  at  384  Kbps  visually  shows  considerable  deterioration  at  higher  temporal 
frequencies,  particularly  in  the  bottom  of  the  picture.  Figure  5.2  shows  the  resulting  two 
branches  of  the  temporal  response  curve. 

The  printout  of  the  frame  repetition  rate  gives  the  average  of  the  spacing  of  the 
points  of  temporal  advance  of  the  test  pattern.  This  number  is  always  essentially  2  for 
codec  H  but  has  higher  values  at  the  other  codecs,  increa&ing  at  lovi/er  bit  rates  and  with 
the  challenge  of  the  test  pattern.  The  frame  repetition  rate  which  is  adaptive  for  most  low 
bit  rate  codecs  is  significant  for  two  reasons.  When  it  becomes  high  enough  to  approach 
the  number  of  frames  per  spoke  as  given  on  Table  4-2,  the  measured  amplitudes  will  be 
distorted,  and  it  is  possible  that  one  spoke  (either  black  or  white)  may  be  skipped  in  the 
analysis.  This  phenomenon  may  be  called  temporal  aliasing  and  gives  false  results  beyond 
the  limit  where  the  two  parameters  are  approximately  equal.  Another  application  of  frame 
repetition  rate  is  covered  in  the  subsequent  paragraph. 

Though  families  of  curves  like  the  ones  on  Figure  5.1  are  descriptive  of  the  motion 
rendition  capabilities  of  a  codec,  a  reference  is  necessary  to  put  the  plotted  values  in 
perspective  and  establish  numerical  limits  of  acceptable  performance.  Therefore  the  RMS 
values  for  all  23  test  patterns  before  processing  through  a  codec  were  computed  in  the 
same  fashion  as  above.  For  an  ideal  static  square  wave  extending  between  1 0  and  1 00 
IRE  units  this  value  theoretically  should  be  45  but  limitations  in  dynamic  response  and 
minor  random  deviations  produce  lower  numbers.  The  mean  values  for  the  3  spoke  widths 
are  as  follows: 
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TEMPORAL  RESPONSE  SUMMARY  -  CODEC  H 
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TEMPORAL  RESPONSE  SUMMARY  -  CODEC 


TEMPORAL  RESPOHSE  SUMMARY  -  CODEC  P 


AMPL 


30  Degree  Spokes:  41 .3 

18  Degree  Spokes:  39.9 

10  Degree  Spokes:  35.4 

The  difference  between  these  values  and  the  RMS  amplitudes  at  the  codec  output 
provides  a  measure  of  motion  performance  degradation  produced  by  the  codecs. 

5.2  Related  Motion  Parameters 
5.2.1  Jerkiness 

All  codecs  reduce  the  transmission  bit  rate  requirement  by  reducing  the  transmitted 
frame  rate  and  repeat  frames  at  the  receiver.  This  produces  jerkiness  in  the  received 
picture  which  becomes  more  pronounced  as  the  transmitted  frame  rate  decreases  and  the  ‘ 
number  of  repeated  frames  increases.  Since  the  frame  repetition  rate  is  one  of  the  results 
of  the  analysis  of  the  motion  test  patterns  a  direct  measure  of  jerkiness  is  obtained  as  a 
by-product  of  the  temporal  frequency  response  measurement.  Due  to  the  adaptive  frame 
repetition  feature  of  most  low  bit  rats  codecs,  jerkiness  increases  as  the  bit  rate  is  reduced 
and  the  test  pattern  becomes  more  challenging.  Though  each  frame  repetition  must  be  a 
whole  number,  averaging  over  several  frames  generally  produces  a  fractional  value.  The 
transmitted  frame  rate  is  obtained  by  dividing  30  by  the  frame  repetition  rate. 

The  results  of  the  jerkiness  measurements  for  codecs  L  and  P  are  given  in  Tables  5- 
6  and  5-7.  Codec  H  has  a  fixed  frame  repetition  rate  of  2  (transmitted  frame  rate  of  1 5 
FR/sec)  which  is  part  of  the  equipment  specification,  therefore  the  measurement  result 
would  be  trivial.  All  adaptive  frame  repetition  rates  of  codecs  L  and  P  are  averages 
rounded  to  the  nearest  whole  number.  The  values  decrease  with  increasing  bit  rate  and 
increase  only  somewhat  with  more  challenging  test  patterns.  The  values  for  the  two 
codecs  are  noticeably  different  due  to  different  design  trade-offs  used  by  the 
manufacturers.  Examination  of  Tables  5-4  and  5-6  versus  5-5  and  5-7  shows  that  codec  L 
is  designed  for  less  jerkiness  at  some  sacrifice  of  temporal  frequency  response  when 
compared  with  codec  P. 
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JERKINESS  SUMMARY  -  CX>DEC  L 
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JEBKINBSS  SUMMARY  -  CODEC  P 


5.2.2  Image  Update  Time 

In  videoconferencing  and  videophone  systems  there  is  typically  an  "update"  mode 
of  operation  when  the  output  image  must  change  to  a  totally  new  picture  as  rapidly  as 
possible.  For  example,  this  mode  occurs  when  a  scene  cut  is  introduced.  In  this  case,  an 
artifact  appears  which  is  particularly  visible  at  low  bit  rates.  The  artifact  may  take  the 
form  of  blocking,  blurring,  retention  of  the  previous  picture,  etc. 

Measurements  of  image  update  time  were  performed  on  codecs  H  and  L  using  the 
scene  cut  patterns  A-1,  A-13,  and  A-25  as  shown  on  Table  4-1  and  the  test  setup  of 
Figure  4. 1 .  Switching  from  an  all  black  screen  to  the  white  circle  pattern  with  a  switching 
interval  of  4  seconds  (120  frames)  gave  the  best  results.  Update  of  the  image  occurs 
usually  in  2  or  3  discrete  steps,  and  in  some  cases  there  are  distinct  differences  between 
top  and  bottom  of  the  picture.  Therefore,  measurements  were  made  on  lines  No.  25,  150, 
and  255  whenever  needed. 

The  measurement  results  are  summarized  in  the  six  simple  graphs  on  Figure  5.3. 
The  update  time  is  defined  as  the  number  of  frames  necessary  after  switching  to  achieve 
100%  output  level.  This  definition  gives  the  widest  range  of  useful  data  which  are  fully 
consistent  in  their  relation  to  other  system  parameters.  As  expected,  the  update  time 
increases  at  lower  data  rates  and  with  increasing  challenge  on  the  codec  algorithm 
produced  by  smaller  circles.  Each  codec  has  almost  perfect  performance  at  its  highest 
data  rate. 

A  comparison  of  the  two  tested  codecs  is  possible  at  the  common  data  rate  of  384 
Kbps.  This  was  done  subjectively  during  equipment  demonstrations  when  both  codecs 
were  operating  side-by-side.  The  unanimous  opinion  of  the  observers  was  that  at  384 
Kbps  codec  L  (with  its  algorithm  optimized  for  low  data  rates)  gave  superior  performance. 
The  same  is  brought  out  clearly  in  the  graphs  of  Figure  5.3  This  shows  full  correlation 
between  the  update  time  measurement  and  subjective  evaluation. 
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FIGURE 


6.0  DISCUSSION 

The  results  summerized  in  Section  5.0  show  that  the  requirements  postulated  in 
Paragraph  3.2  have  been  satisfied  by  the  performed  measurements  of  temporal  frequency 
response.  All  measurements  were  performed  using  the  taped  processed  picture  only.  The 
test  signals  are  firmly  defined  and  computer  generated.  Evaluation  of  the  processed  signal 
waveforms  is  straightforward  andrequires  only  standard  video  test  equipment  and  a  simple 
computer  program  to  achieve  numerical  results.  Correlation  with  subjective  evaluation  will 
make  it  possible  to  establish  numerical  limits  of  acceptability  for  many  different 
applications  of  digital  video  technology.  Since  a  large  number  of  motion  artifacts  produce 
distortions  in  video  amplitude,  temporal  frequency  response  provides  a  measure  of  the 
combined  affect  of  these  artifacts. 

A  review  of  the  consolidated  test  results  shows  the  following  relations  between  the 
various  operation  and  test  parameters: 

0  Temporal  response  decreases  with  increasing  temporal  frequency 

0  Temporal  response  decreases  at  lower  bit  rates 

0  Temporal  response  decreases  with  a  "busier”  test  pattern,  meaning  narrower 
spokes  or  smaller  and  denser  white  circles. 

These  results  are  logical  and  exactly  according  to  expectations.  Limited  subjective 
observations  also  have  shown  good  agreement.  These  facts  verify  not  only  the 
methodology  of  measuring  temporal  frequency  response  but  primarily  the  validity  of  this 
parameter  as  descriptive  of  the  overall  performance  of  a  digital  video  system.  One  single 
parameter,  temporal  f  equency  response  which  is  descriptive  of  the  overall  visual 
impression  on  an  average  viewer  is  the  most  useful  tool  for  objectively  evaluating  the 
motion  performance  of  a  digital  video  system.  This  will  be  confirmed  by  correlation  of 
subjective  and  objective  test  results  which  will  be  performed  on  a  later  program. 
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7.0  CONCLUSION  AND  RECOMMENDATIONS 

The  effort  described  in  this  report  has  proved  that  Temporal  Frequency  Response  is 
a  valid  practical  parameter  to  describe  the  effect  of  most  of  the  motion  artifacts  of  a  digital 
video  codec.  The  proposed  measurement  methodology  yields  additional  parameters  as  by¬ 
products.  It  was  shown  that  the  previously  designed  test  patterns  were  adequate  to  test 
codecs  over  the  full  range  of  bit  rates  from  64  to  1 536  Kbps.  The  convenience  of 
measurement  and  accuracy  of  the  results  were  improved  by  choosing  a  different  location 
of  the  measurement  point.  A  survey  of  the  latest  available  test  equipment  and 
procurement  of  the  applicable  items  made  it  possible  to  use  3/4"  video  tape  instead  of  the 
previously  needed  1 "  tape.  This  resulted  in  a  considerable  simplification  and  reduced  cost 
of  the  test  procedures. 

The  test  results  show  excellent  consistency  with  anticipations  and  logical 
interdependency  of  the  various  parameters.  There  also  is  good  correlation  with  limited 
informal  subjective  impressions  but  final  verification  of  the  methodology  must  await 
correlation  with  the  results  of  formal  subjective  tests  which  are  being  initiated  under  a 
concurrent  program. 

The  measurement  results  show  that  the  status  of  objective  motion  testing  is  well 
advanced.  Though  some  refinements  of  the  existing  methodology  are  expected  to  be 
possible,  the  next  big  progress  should  result  in  the  elimination  of  the  requirement  for  a  test 
tape  and  the  partly  manual  evaluation.  An  electronic  pattern  generator  with  a  wide  variety 
of  selectable  motion  test  patterns  can  and  should  be  designed.  Similarly,  the  evaluation  of 
the  processed  patterns  can  be  automated,  eliminating  the  need  for  taking  and  recording 
visual  readings  from  the  VM-700  display.  Both  these  projects  will  require  a  considerable 
design  effort. 
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